Touch panel input assisting device, computer operating method using the device, and tactile sense interlocking program

ABSTRACT

[PROBLEMS] To detect any of various movements of a finger by using the touch panel of an existing device and to give tactile sense to facilitate operation. 
     [MEANS FOR SOLVING PROBLEMS] A touch panel input assisting device attached to an existing device having a touch panel or held by fingers so as to use it. Any of various movements of a finger is transformed into rolling of a rolling unit on a touch panel or deformation thereof, the rolling or deformation is detected as a variation of the position or area of the contact of the rolling unit with the touch panel, and a tactile sense such as a reactive force, a resistance force, vibration, or a sensation of roughness is given to the finger. A tactile interlocking program for analyzing the movement pattern and speed pattern of the point of contact of the rolling unit with the touch panel and estimating the movement of the finger is used to perform a predetermined operation interlockingly with the movement of the finger or tactile sense given to the finger.

TECHNICAL FIELD

The present invention relates to a touch panel input assisting devicethat converts various finger movements into contact patterns by arolling motion of a rolling unit on a touch panel while giving reactionforce, such as a resistance force or repulsive force to the finger; anda computer operating method to detect the timing of the original fingermovements or the reaction force being exerted to the finger based on thechange patterns of the position or speed of a contact point of therolling unit on the touch panel and to execute a computer bysynchronizing the timing with the computer and a tactile senseinterlocking program.

BACKGROUND ART

Conventionally, a finger or a pen-like-instrument called a stylus hasbeen used to contact the touch-panel surface, and to input coordinatesof the contact point. However, in either case, tactile feedback has beeninsufficient and, as a result, visual or auditory sense has been neededto confirm whether the input was performed correctly.

In order to cope with this problem and enable it to confirm the resultor content of the input, such as the contact position etc. not onlythrough vision or hearing but through a tactile sense, this inventionprovides an apparatus that lies between the finger and touch panel toassist the touch panel operation and a method or a program that uses theapparatuses effectively in aiming for improvement of the usability ofthe user interface or the reality of the game operation. Althoughvarious kinds of tactile feedback devices have been developed, the trialof solving the above-mentioned problem through a device which isinterposed between a touch panel and a finger does not exist, and thatis the point of the novelty of this invention.

The patent documents relevant to this invention are shown below.

Patent Document 1: Japanese Unexamined Patent Application PublicationNo. 9-128149

Patent Document 1: Japanese Unexamined Patent Application PublicationNo. 2001-306238

The method of inputting data and instructions by the operation ofrolling a roller-like unit on a touch panel is shown in patent documents1 and 2. However, any technique uses the rolling unit only as a meansfor inputting information, and does not use it as a means for addingtactile feedback, and the device for it was not given.

DISCLOSURE OF INVENTION

This invention uses a rolling unit as a means to apply a feeling of notonly unevenness but reaction force to a finger. And by making a programsynchronize with such a tactile feedback, operation of the program canbe confirmed through the tactile sense, and the usability of a userinterface and the presence of a game are improved. Moreover, in thisinvention, a touch panel input assisting device is applied to the touchpanel, which functions alone (independently) and can be operated also bythe finger or a stylus, by equipping the touch panel with the apparatusseparately, by equipping the finger with the apparatus or by holding theapparatus with the finger. Therefore, the means to equip the touch panelwith the apparatus so that the attachment and detachment are possible,or the holding or the wearing means for the finger to be equipped withthe apparatus is established in this invention. Thus, it became possibleto exclude the sensing function by the side of thetouch-panel-input-assisting-device, and to reduce the device costsharply by detecting movement of a finger utilizing the sensing function(i.e. contact position detection mechanism) of the existing touch panel.Moreover, the structure for equipping a finger with thetouch-panel-input-assisting-device is established in one of embodimentsof this invention. Such a concept of having equipped a finger with thetouch-panel-input-assisting-device and operating a touch panel with itis new and does not exist previously. Although there may be somestructure which can be interpreted also as a type of rolling unit,rolling on the position detecting sensor in the conventional joy stick,as it contains sensing function equipment in itself, its structurebecomes complicated and the cost also becomes high, unlike thisinvention.

This invention was made in view of the above situation and uses aspecial instrument interposed between a finger and a touch panel inwhich the tactile feedback lacks for its operation, and thereby it givesa tactile feeling of the reaction force or the uneven surface to thefinger under the operation and, by controlling a program to synchronizewith these tactile feeling, it improves the usability of a userinterface and the presence of a game and offers thetouch-panel-input-assisting-device, which can be produced at a low costby utilizing the sensing function of a touch panel, a computer operatingmethod using the device, and a program synchronized with the tactilefeeling. A means for giving reaction force to a finger is hereaftercalled a “reaction-force-generation-means”.

In order to achieve the above-mentioned object, an element called arolling unit is interposed between a finger and a touch panel in thisinvention. And by using it, various finger movements are converted intothe rolling movements of the rolling unit on a touch panel, and then thecontact position of the rolling unit on the touch panel and its change(that is, consequently the contact patterns generated by the rollingmovements) are detected using the contact position detecting function ofthe touch panel, and thereby the original finger movements and thereaction force applied to the finger during the finger movements aredetected. Moreover, the reaction force such as a resistance force or arepulsive force makes it possible for the operator to confirm theappropriateness of the finger movement tactually, and thus the operatorcan adjust the quantity of motion or the degree of force by using thetactile feedback and input analog information with sufficient accuracy.A means to convert the various finger movements into the various contactpatterns, which a touch panel can detect, will be hereafter called acontact-pattern-generation-means. The contact pattern means the contactform, such as a position or an area size where thetouch-panel-input-assisting-device contacts a touch panel, or the formof their change with regard to time. Although a pattern has the meaning“model” besides the meaning “form”, this invention uses it in the senseof the latter.

Conventional touch panels only detect the contact to a touch panel, theseparation from the touch panel, and the movement of a finger or astylus to a direction parallel to the panel surface. However, accordingto this invention, the finger movement perpendicular or slanted to thetouch panel surface is detectable by converting the finger movement intothe movement of a rolling unit on the touch panel. Moreover, movement ofthe finger for rotating a sphere-like-object such as a trackball, or adial is detectable by converting it into the movement of a rolling uniton the touch panel. Moreover, since the contact point of a rolling unitand a touch panel will move by a characteristic pattern specific to eachfinger movement pattern, the original finger movement pattern can beestimated by analyzing the pattern by a program. In addition, theconventional input operation using a finger or a stylus on the touchpanel were not able to make the operator feel the tactile feedback suchas a resistance force, reaction force, a feeling of unevenness, orvibration. However, using a rolling unit makes it possible to give suchtactile feedback.

According to this invention, upon using a device equipped with anexisting touch panel, an operator can use the separate and low-costtouch-panel-input-assisting-device, which is attached to the device or afinger or held with a finger, and thereby the operator can feel thetactile feeling, which his finger would feel when operating otherfamiliar devices, and input information correctly and intelligibly. Inaddition, by using the tactile sense interlocking program whichestimates the movement of the finger from the movement pattern of thecontact position of a rolling unit and a touch panel, the deviceoperation can be synchronized with the finger movement and the tactilefeeling and it becomes possible to have presence while operating thedevice.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 is a functional block diagram of a computer system 89 which usesa touch-panel-input-assisting-device 80 as a data input device. Thecomputer system 89 includes an input device 85 which inputs data, adata-processing device 87 which performs the data processing of aprogram using the inputted data, an output device 86 which outputs theresult of the processing, and a storage device 88 which stores data.Moreover, the input device 85 includes a touch panel 2 which consists ofa panel 81 and a contact-position-detection-means 82, and thetouch-panel-input-assisting-device 80 which inputs data to the device 80through this touch panel 2. The computer system 89 can be ageneral-purpose personal computer or may be a game machine or a musicplayer for exclusive use. The touch-panel-input-assisting-device 80 ofthis invention is used in combination with the panel 81 equipped withthe contact-position-detection-means 82 prepared separately from it asshown in FIG. 1. Hereafter, the panel equipped with thecontact-position-detection-means will be called the touch panel 2. Thistouch panel 2 is given by the existing equipment, separate from thetouch-panel-input-assisting-device of this invention. Moreover, atactile sense interlocking program of this invention is stored in thestorage device 88 of FIG. 1 and executed by a data-processing device 87.The tactile sense interlocking program detects the movement of a fingerwhich operates the touch-panel-input-assisting-device 80, based on theinformation given by the contact-position-detection-means 82 about thecontact position and, if necessary, the size of contact area of thetouch panel and the touch-panel-input-assisting-device 80.

The tactile sense interlocking program is stored in the storage device88, loaded to the data-processing device 87, and is executed. Forexample, the movement or speed pattern of the contact position of arolling unit and a touch panel is compared with the patterns registeredin advance, and a predetermined operation which is related to thematched pattern is executed. More simply, when a rolling unit separatesfrom the touch panel, the operation associated with the amount or thespeed of movement of the contact point is executed. The time when theoperation is executed can be the time when the moving direction of thecontact point is being reversed, or when a predetermined position,distance or speed is reached. In this case, since the contact positionand its speed generated by the contact-pattern-generation-means arerestricted to the range between the lower and upper limits which arecharacterized by the touch-panel-input-assisting-device, it is necessaryto define the mapping relation between the pattern and the operation inconsideration of the characteristic of thetouch-panel-input-assisting-device 80, so that the operation needed canbe executed within the value of the restricted range.

(The Operation Method of Mapping Relation)

The contact pattern which the contact-pattern-generation-means of thetouch-panel-input-assisting-device 80 generates is detected by thecontact-position-detection-means 82 of a touch panel, for example as atime series signal of the amount of change ΔX, movement speed V, movingdirection θ of a contact position. When the finger holding thetouch-panel-input-assisting-device 80 performs a predetermined action,the actual values which this time series signal takes depends on thecharacteristic of the touch-panel-input-assisting-device 80 and a touchpanel. For example, the signal ΔX, which thecontact-pattern-generation-means generates when thetouch-panel-input-assisting-device is operated so that thereaction-force-generation-means of thetouch-panel-input-assisting-device 80 may apply a properly strongreaction force to a finger, might be too small and the amount ofoperation might be insufficient when the small signal value is used asit is for the amount of operation of the program. In that case, scaleadjustment by multiplying the constant α is needed so that the value ofthe proper range may be obtained by the following formula to define theamount of operations of the program, for example, where ΔZ is the amountof movement of a character in a video game. ΔZ=α·ΔX (formula 1 whichdefines the mapping relation of a contact pattern and the amount ofoperation of the program).

Moreover, as for the moving direction 8 of the contact point, thereference direction S needs to be determined according to thecharacteristic of the touch-panel-input-assisting-device 80 and thetouch panel 2. In order to take a proper and intuitive correspondencebetween the direction of the finger movement or the direction of thereaction force and the direction O of a character movement, it isnecessary to define the mapping relation between them by the followingformula for example. O=θ−S (formula 2 which defines the mapping relationof a contact pattern and the amount of operation of the program).

When the touch-panel-input-assisting-device 80 separates from the touchpanel 81 after contacting the touch panel 81, thereaction-force-generation-means applies reaction force of a properstrength to a finger. The timing of the separation or the forceapplication is detected by the program, checking if each of ΔX and theaverage Vm of the past time series of the speed V exceeds a thresholdpredetermined respectively and the direction of speed is reversed. Inthis case, the mapping relation between a variable F, which defines theoperation of program, and ΔX, V, Vm, and θ of a contact pattern isdefined in the following procedure of the program.

-   IF ΔX>β AND Vm>γ AND θ=180 degrees THEN F=TRUE ELSE F=FALSE    (procedure which defines the mapping relation between the contact    pattern and the operation of the program).-   In order for the touch-panel-input-assisting-device 80 to give    suitable tactile feeling for a finger and in order to give a    suitable margin to avoid malfunction, the threshold values of β or γ    should be adjusted according to the characteristic of the    touch-panel-input-assisting-device 80 and the touch panel 2.

Furthermore, the contact pattern (ΔX, V, θ) and its time series signal[(ΔX(1), V(1), θ(1)), (ΔX(2), V(2), θ(2)), . . . ] for a predeterminedfinger movement with a predetermined tactile feeling are paired with thecorresponding operation of the program and registered in a table inadvance. (The number in a parenthesis means the time when each signalwas detected). The contact pattern which the touch panel 81 detects iscompared with the patterns registered in the table, and if there is aregistered pattern which matches the contact pattern, the operation ofthe program paired with the registered pattern can be executed. Althoughthis is an example which realizes mapping relation with a definitiontable, the pattern registered is defined using thetouch-panel-input-assisting-device. (As a result, the registered patternis dependent on the characteristic of thetouch-panel-input-assisting-device.) Since such parameters as α, β, γ,and S and the registered patterns are determined so that the fingermovement, the reaction force and the operation of the program aresynchronized with a proper amount and a proper timing, they are calledthe data for synchronization. The data for synchronization is memorizedin the procedure of computation or in the memory of a computer inadvance and, by using them, the tactile sense interlocking program canacquire the key for a synchronization from the contact pattern itselfwithout additional signal for synchronization, and it can synchronizethe operation of the program with the finger movement or the tactilefeeling of the finger.

FIG. 11 shows the features of the synchronous method of this inventionthat synchronizes the reaction force which the finger feels (reactionforce which a reaction-force-generation-means applies to a finger whenthe touch-panel-input-assisting-device is operated with a finger) withthe operation of a computer or a program using the data forsynchronization. For comparison, the conventional synchronous methodusing the signal for synchronization is also shown. Among the variousabove-mentioned mapping systems, as an example, the case where mappingrelation is defined by the easiest formula is illustrated forsimplicity. As for the case where mapping relation is defined by aprocedure, the map in the figure can be defined by the procedure, andthe data for synchronization can be represented by the parameters andthe registered patterns which are used in the procedure.

FIG. 11 shows an example of the computer operation method or the tactilesense interlocking program of this invention in (a), and an example byconventional method in (b). In both of the examples, thetouch-panel-input-assisting-device is provided as a peripheral deviceout of the computer. And independently from thetouch-panel-input-assisting-device, the computer operating method or thetactile sense interlocking program of this invention is implemented inthe computer. Moreover, the touch panel is separate from thetouch-panel-input-assisting-device and the contact pattern generated bythe touch-panel-input-assisting-device inputs into the computer throughthe touch panel. Although the information exchanged between thetouch-panel-input-assisting-device and the computer is only the contactpattern in (a), the signal for synchronization also needs to beexchanged besides the contact pattern in (b). The most important featureof (a) is that, by memorizing the various characteristics (such as therange of values of the contact pattern for the proper movement range ofa finger or the value of the contact pattern for reaction force of apredetermined strength) of the touch-panel-input-assisting-device in thememory in the computer or to the parameters of the procedure as data forsynchronization in advance before computer operation, it can estimatethe timing of the finger movement performed and the timing of thereaction force generated only from the information on the contactpattern, without signal for synchronization, and operate the computersynchronizing with the timings.

The map in FIG. 11( a) is defined by the formula of ΔZ=αΔX, where ΔX isthe amount of displacement of the contact point on the touch panel givenby the input operation to the computer, and ΔZ is the amount of movementof an object on a screen of a video game. When playing a game, there isa proper range for the finger movement to control the game and thereaction force should be applied to the finger within a proper range ofstrength. In order to move the object on the screen of the video gamewith a proper amount by the operation within those proper ranges, thevalue of α must be determined appropriately. And for this purpose, thecharacteristic of the touch-panel-input-assisting-device must beinvestigated in advance, and it must be recorded in the memory as datafor synchronization. This characteristic includes the range of values ofthe contact pattern generated by the touch-panel-input-assisting-devicewhen it is manipulated under a proper range of finger movement, or aproper range of reaction force. Since the operation of the program canbe controlled only from the contact pattern without any signal forsynchronization by using the value of the a in the computer side, α iscalled the data for synchronization.

When operating the touch-panel-input-assisting-device of FIG. 3 with afinger, the range of finger movement for a proper range of strength ofreaction force or the range within which a finger can move comfortablyis constrained inside the ellipse of the dashed line of 206 in FIG. 12.Moreover, the ellipse of the dashed line of 207 shows the range withinwhich the contact point of the rolling unit 31 on a touch panel movesfor the movement range of the finger. The range of 207 is determined bythe characteristic of the touch-panel-input-assisting-device, and itbecomes the range of half the movement range of a finger in thisexample. The value of α must be determined so that, according to theamount of displacement of the contact point generated within this range,the object on the screen can move with a proper amount. Thetouch-panel-input-assisting-device of FIG. 12 differs a little from whatwas shown in FIG. 3. In the example of FIG. 3, thetouch-panel-input-assisting-device is equipped to the finger using aband 42 tied around the finger and fastened with the buckle 43, while inthe example of FIG. 12, the touch-panel-input-assisting-device isequipped to the finger using a band that is fastened by the restorationforce generated by the thick rubber part 205 which pulls the band whenit restores to a straight state.

In this invention, the panel equipped with thecontact-position-detection-means is generically named the touch panelregardless of the kind of the detection means. When acontact-position-detection-means is a pressure-sensitive type like aresistive film sensor, the contact area of a rolling unit on a touchpanel needs to be small and localized so that a proper amount ofpressure can be applied. On the other hand, when acontact-position-detection-means is an capacitive sensor, the rollingunit or the film which is interposed between a rolling unit and a touchpanel needs to be made of the material with proper conductivity ordielectric constant, and a part of it needs to be in contact with thehuman body or the reference electrode of the sensor. Furthermore, when acontact-position-detection-means is an optical sensor, the rolling unitsurface needs to be equipped with a marker to be identified, or it isnecessary to consist of materials which may reflect the modulated lightwhich is emitted from a light source.

Alternatively, a touch panel is comprised of the electrodes arranged inthe form of a matrix so that a pressured position may be detected by thecontact relation between electrodes, and the contact position at whichthe rolling unit rolling on the panel surface contacts directly orindirectly to the panel is detected. When contacting directly, the lowerpart of the rolling unit contacts directly to the surface of a touchpanel, however, when contacting indirectly, it presses in-between sheetor belt which were made of the flexible material like rubber or cloth,and the sheet or belt pressed by the rolling unit contacts the touchpanel 2. For example, in the example of composition of FIG. 2, thesilicone rubber sheet 50 is stretched over the touch panel 2 with aproper gap separating it from the surface of the touch panel 2. Thesilicone rubber sheet is deformed at the position where it is pressed bythe rolling unit and contacts the touch panel. As the rolling unit rollspushed by the finger, the sheet is pressed at different positions andthe coordinates of the contact positions (51, 52) inputs.

The sectional view which cuts a ball-like rolling unit 31perpendicularly is shown in FIG. 2. A cylindrical hole is formed at thebottom of the ball-like rolling unit 31, and a spring 44 is placed init. The upper end of the spring is attached to the upper part of thecylindrical hole with a metal connector 45. Moreover, the lower end ofthe spring is attached to a silicone rubber sheet 50 through the jointcomponent 41. Using a support component 53 as a spacer, the siliconerubber sheet 50 is stretched over the touch panel 2 separated with aproper gap from the panel surface. If the upper part of the ball-likerolling unit 31 is pushed down with a finger, the silicone rubber sheetis deformed as shown in (b), and the ball-like rolling unit lower partsandwiches a silicone rubber sheet and the joint component 41in-between, and contacts the touch panel 2 at the position of 51.Moreover, when the finger moves leftward on the figure, pushing theball-like rolling unit 31 against the touch panel 2 with a finger asshown in (c), with the spring 44 being extended, the ball-like rollingunit 31 rolls towards the left, and contacts the touch-panel surface inthe position of 52 through the silicone rubber sheet which is pushed anddeformed. Meanwhile, the spring 44 tries to restore itself to theoriginal length and the reaction force F1 is applied to a finger to thedirection indicated by an arrow in the figure, and thus this mechanismexplained above serves as a reaction-force-generation-means. Thestrength of this reaction force can be computed based on the contactposition 52 on a touch panel. In addition, an elastic silicone rubber 77can be used instead of the spring 44 as shown in FIG. 5 and it can bemolded as a single piece with the sheet 50 which is also made fromsilicone rubber. As shown in (c), the tip part 78 which is thinner thanthe main part 77 can be pulled upward after letting it pass in the holeof the ball-like rolling unit 31 made from silicone rubber. By pulling78 upward, a swelled portion 76 of 77 can be drawn into the cavernouspart 75 inside the ball-like rolling unit 31, jointing 77 and 31 asshown in (d), cutting 78 with the dashed line 79 of (d) after that, andthe structure shown in (a) is producible simply and at a low cost.

The length of the spring 44 is secured by connecting it to the bottom ofthe concavity of a rolling unit, letting the stress added to thematerial distributed when making the rolling unit operate and roll.Moreover, if the material of the rubber sheet 50 or the rolling unit 31is made from conductive rubber, the rolling movement of the rolling unitis detectable with a capacitive type touch panel. If they are made fromconductive materials such as conductive rubber in other embodiments ofthe invention hereinafter, the contact pattern is also detectable with acapacitive type touch panel.

The touch-panel-input-assisting-device 80 by this invention contacts atouch panel through a rolling unit. Even when it is equipped to thefinger, a rolling unit is connected through a spring or an elastic cord,for example, so that the elasticity of them may draw the rolling unit toa finger and have the flexibility with which the rolling unit can rollon the surface of the finger. The example of composition of such atouch-panel-input-assisting-device 80 is shown in FIG. 3. The ball-likerolling unit 31 has the same structure as FIG. 2 and is attached to aband 42 a through the joint component 41 using the same principle asFIG. 2, where the band is tied around the finger. After tied with abuckle 43 and passing through it, although not contained in a figure,the part of the band 42 b is further extended, its end leads to thehousing of the touch panel 2, connected with it, and functions as astrap which prevents fall and loss of thetouch-panel-input-assisting-device 80. The side view which looks at (a)from the thumb tip side is shown in (b).

The section of the ball-like rolling unit of FIG. 3 and the principle ofrolling operation are shown in FIG. 4. The side view seen from the tipside of the thumb is shown in (a). It shows the section when theball-like rolling unit 31 is divided perpendicularly where 42 c is theband tied around the thumb, and the end of the spring 44 is attached tothis band through the joint component 41. This spring passes through thecylindrical hole which was made inside the ball-like rolling unit 31,and its end is joined to the bottom of the hole by the metal connector45. The side view seen from the transverse direction of the thumb isshown in (b) where 46 is the position where the rolling unit 31 touchesthe panel 2 before rolling operation. The scene of moving the fingerhorizontally toward the transverse direction and making the ball-likerolling unit roll is shown in (c) and the scene of moving the fingerbackward and making the ball-like rolling unit roll is shown in (d). Inany case, the ball-like rolling unit rolls along with a motion of thefinger, and the contact position moves to 47. Meanwhile, the spring 44is extended and the reaction force which draws a finger back to thedirection F1 is generated when the spring shrinks and is restored. Thiswill function as a reaction-force-generation-means.

The touch-panel-input-assisting-device 80 in FIG. 7 generates thereaction force according to the amount of deformation of the rollingunit 31 when the rolling unit is struck or crushed. For example, if theside of the rolling unit 31 in contact with the touch panel 2 is formedin the shape of a curved surface with a material with flexibility andelasticity, such as silicone rubber, and pressed against the touch panel2 by giving pressure on the rolling unit 31, the above-mentioned curvedsurface will be crushed and will contact the touch panel 2 in a largearea while giving reaction force to a finger. The amount of deformationof the rolling unit, the strength of the pressure given to the rollingunit and the strength of the reaction force given to the finger can becomputed from the size of the contact area on the touch panel 2. It isalso possible to compute the reaction force given to a finger bypreparing pressure sensors, such as a piezoelectric device and detectingthe pressure added to the touch panel 2. The contact area size of therolling unit 31 on the touch panel 2 is detectable through theresistance or the electric capacity value which the sensor on the sideof the touch panel 2 detects, or through the number of electrodes whichcontacts each other electrically. The touch-panel-input-assisting-device80 based on such a principle is shown in FIG. 7. A circular section 55shows a section of the ball-like rolling unit 31 mentioned above. If therolling unit is formed with the material such as a hollow rubber, whichis easily deformed, in any case, the contact area size of the rollingunit 31 on the panel 2 at the contact position 56 is small in the caseof being pushed against the panel 2 by a weak force as shown in (a), andlarge in the case of the crushed and deformed rolling unit being pushedagainst the panel 2 by a strong force as shown in (b) for the contactposition 57. Meanwhile, in (b), the crushed rolling unit 31 tends to berestored to the original form, and generates the reaction force F2. Thestrength of this reaction force can be computed from the size of thecontact area of 57 by measuring it by a certain method.

For example, in the case of the resistive type touch panel sensor, whencontacting the touch panel 2 by two points simultaneously, the position(center-of-gravity position), which is the average over the contactpositions with the weight of the contact area size, is detected as acontact position. For example, in FIG. 7, P and Q are detected as thecenter-of-gravity position. When the size of the contact area changesaccording to the degree of deformation of the rolling unit, thecenter-of-gravity position of the two points changes according to thedegree of deformation. Therefore, the degree of deformation can bedetected from the center-of-gravity position which the touch paneldetects.

In the touch-panel-input-assisting-device 80 described so far, in anycase, the rolling unit rolls, changing the contact position on thefinger surface as well as the contact position on the panel surface,being sandwiched between the two surfaces, if a finger moves pushing arolling unit against the touch panel 2. The amount of rotation of therolling unit can be computed from the contact position detected by thetouch panel. During this operation, the spring or the elastic cordconnecting the rolling unit are extended and the reaction force isgenerated when it is restored to the original length. Since the strengthof this reaction force is computable through the amount of movement (theamount of change) of the contact position while the rolling unit rollskeeping in touch with the touch panel, the program can be controlled tosynchronize with the reaction force. Although the reaction force canalso be computed directly by detecting the tension of the string whichpulls a rolling unit using a force sensor or a distortion sensor, etc.instead of computing the reaction force indirectly from the amount ofmovement of the contact position, it is desirable to compute thereaction force through the amount of movements of contact position as itis detectable on the side of the touch panel.

In the tactile sense interlocking program of this invention, byassociating (relating or mapping) the reaction force computed asmentioned above with a variable in the program, it becomes possible tograsp the contents of operation through the reaction force, or thepresence of a game can be improved. That is, when an operator rolls arolling unit, he can grasp the value of the variable through thestrength of the reaction force which a finger feels, and the directionof a vector in the program through the direction of reaction force. Itis also possible to associate the direction angle of the reaction forcewith the value of a variable if the direction is represented by an angleinstead of a vector. For example, in a video game of shooting a target,the strength of the reaction force can be associated with the strengthof power with which a ball is shot and the direction of the reactionforce can be associated with the direction to which a ball is shot. Withthese associations, the force which a finger feels and the force appliedto the object in a game are intuitively related, and the presence of theshooting game is improved. In the games such as ‘Pachinko’ (Japanesepinball machine) for which the feeling of the reaction force felt whenpulling or releasing an elastic string or the feeling of bouncing forcewhen pushing an object is important, the reaction force which thetouch-panel-input-assisting-device 80 of this invention generates canimitate those forces and improve the reality or presence of the games.

The tactile sense interlocking program is a program used in combinationwith the touch-panel-input-assisting-device and executes a predeterminedoperation synchronizing with one of the following events, the rollingunit separates from the panel, the contact position of the rolling unitand the panel moves for a predetermined distance or arrives at apredetermined position, and the moving pattern or the speed pattern ofthe contact position matches a predetermined pattern. Since the positionor the speed of the contact point generated by thecontact-pattern-generation-means is limited within a certain range(between the lower and the upper bounds), it is necessary to define themapping relation between the generated pattern and the predeterminedoperation so that the operation can be performed within the value of thelimited range. (Namely, it is required to taking into consideration thecharacteristic of the touch-panel-input-assisting-device.)

The tactile sense interlocking program is a program which detects thefinger movement which is converted to the rolling movement of therolling unit on the touch panel by thetouch-panel-input-assisting-device, based on the movement of the contactpoint of the rolling unit on the touch panel and the change of themovement speed. This program also presumes the concrete form of thefinger movement if needed, and executes a predetermined operationsynchronizing with movement of the finger. For example, when detecting afinger movement vertical to the touch panel, if the movement of thecontact position of a rolling unit and a touch panel stops afterexceeding the speed of a certain predetermined size, and then reverses adirection, the program can judge that the finger contacted the touchpanel after it approached the touch panel, and stood still on the touchpanel, and then separated from the touch panel. And if the program ismade to perform a predetermined operation at the time of the contactposition movement stopped or at the time of the moving directionreversed, the movement of the finger and operation of the program can besynchronized at these timings. Or more simply, a predetermined operationcan be performed synchronizing with the time of: contact having beencompleted, the contact position arriving at a predetermined position,the movement speed reaching a predetermined speed, or the amount ofrelative movements becoming predetermined amount. Or more generally, theprogram can perform a predetermined operation synchronizing with thetime of the moving pattern (the pattern of movement or speed) of thecontact position matching with the predetermined movement patternregistered in advance. If the reaction force which a rolling unitreturns to a finger changes according to the amount of roll, the amountof reaction force can be estimated by measuring the amount of movementof the contact position while the rolling unit and the touch panel hasmaintained the contact state. So the program is designed to performoperation according to the size of the reaction force synchronizing withthe time of the rolling unit separating from the touch panel. When thesurface of the rolling unit is uneven or having some corners, thecontact position of the rolling unit surface and a panel changesdiscontinuously when a rolling unit is made to roll. So, when adiscontinuous change is detected through a touch panel, it judges thatthe movement of the finger with which a specific position on the surfaceof a rolling unit contacts the touch panel and rolls on it wasperformed, and the program is made to perform a predetermined operation.

For example, the scene of the game in which a ball 61 is hit with a bat60 is shown in FIG. 8. If the correspondences are taken between theangle of the bat, the amount of swinging the bat 62, or the power ofhitting the ball and the amount of roll of the rolling unit or thestrength of reaction force of the touch-panel-input-assisting-device 80,the player of a game can grasp the angle of pulling the bat or thestrength of power to hit the ball through the reaction force given tothe finger. That is, since the power in a virtual game can be grasped byactual power, the presence of the game increases. Moreover, even if theplayer does not gaze at the angle of the bat in a screen, the angle ofthe bat and the strength of power to hit the ball can be tactuallygrasped. As for reference numeral 63 showing the locus of a ball, thespeed of the ball or the power of the bat with which the ball reboundscan be determined according to the speed and the power felt when pullingand releasing a rolling unit. Moreover, whether a ball and a bat collidecan be determined based on the strength of the reaction force added tothe finger just before releasing a rolling unit and the releasing time.The releasing time and the strength of the reaction force are detectableby the contact state and contact position detecting function of a touchpanel. In this case, a suitable mapping relation have to be definedbetween the variable value which defines the angle of the bat in aprogram and the reaction force or the contact pattern so that a properrelation can be established between the angle of the bat and thereaction force added to the finger or the amount of finger movement.Therefore, the program must be designed depending on the characteristicof the touch-panel-input-assisting-device.

In another example, the variable in the program is associated with(related to) the amount of deformation or the reaction force whenapplying pressure and deforming a rolling unit as shown in FIG. 7. Forexample, if a scene of hitting or crushing an object etc. is containedin the contents of a play of a game such as a boxing game orWhack-A-Mole game, the amount of deformation or reaction force when arolling unit is pressed or crushed is detected, and it is related withthe value of the variable which defines the damage given to the opponentor the mole, with this association, the contents of the game and theoperation given to the rolling unit are intuitively related, and thepresence of a game is improved. It is necessary to give suitable mappingrelation between the amount of deformation, the strength of reactionforce, and the variable values specifying such damage in the program.Therefore a program must be designed depending on the characteristic ofthe touch-panel-input-assisting-device.

The touch-panel-input-assisting-device of this invention is equipped onthe panel with the contact-position-detection-means, attached to afinger or held with a finger, and is equipped with the rolling unitwhich contacts the panel directly or indirectly, and rolls on it, and ithas a movement-conversion-means to convert the movement of a fingerwhich approaches the panel surface from the direction approximatelyperpendicular to the panel surface into the rolling movement of therolling unit while the rolling unit is contacting the panel. And thecontact-pattern-generation-means is realized by generating a contactpattern by the roll of the rolling unit, and thereaction-force-generation-means is realized by giving a resistance forceor reaction force to a finger moving close to the panel surface via therolling unit rolling on the panel.

Although the conventional touch panel detects movement of a finger or astylus parallel to a touch panel surface, thetouch-panel-input-assisting-device makes it possible to detect thefinger movement approaching the touch panel surface from theperpendicular or a slanting direction. For that purpose, the rollingunit converts movement of a finger into its own rolling movement on atouch panel, and movement of a finger that approaches the panel isdetected as movement of the contact position of the rolling unit and thetouch panel. Contacting indirectly means contacting via interposingfilm, cloth, etc. which is deformed easily between a rolling unit and apanel. Moreover, the rolling unit is any object that rolls or inclines,changing the position of the contact point on a panel regardless of itsshape, so the form of the rolling unit is necessarily neither a spherenor a wheel form. For example, it can be simply an object which inclineslike a seesaw. It will be called a rolling unit as far as the positionof the contact point on a panel moves when it inclines. If the contactsurface on which the contact occurs between the rolling unit and thepanel is curved, the contact position would move continuously when theunit inclines. If the contact surface has angles, such as a gear, thecontact position would move discontinuously, but it is still called arolling unit as far as the contact position with a panel moves when itrolls or inclines. Since the contact position moves discontinuously by apredetermined skip distance, especially when making a surface with anangle roll, the input by such a rolling unit is distinguished from thatof a usual stylus etc.

The touch-panel-input-assisting-device 80 of FIG. 6 can convert themovement of a finger approaching a touch panel into a rolling movementof the rolling unit continuously contacting the touch panel surface. Anexample of a single piece prototype of thetouch-panel-input-assisting-device 80 made by integral molding withrubber is shown in (d), (e), and (f) of FIG. 6. This single pieceversion operates by the same principle as another prototype of thetouch-panel-input-assisting-device shown in (a), (b), and (c) of FIG. 6.The figure in (d) shows the top view, the upper part 32 of the devicecrawls on the upper surface of a finger (a dashed line shows), and theside part 34 of it is used to hold the device by applying the thumb onit. Figures in (e) and (f) shows the side view, and they show how therolling unit 33 rolls on the touch-panel 2 when a finger moves close toa touch panel. The scene of (e) shows the rolling unit 33 contacting thetouch panel 2 at the position of 36. When the finger is brought furtherclose to a touch panel, the rolling unit 33 rolls on the touch panel 2and the contact position moves to the position 37 as shown in (f). Asthe whole device is made as a single rubber piece, the device must bendbetween the rolling unit (rolling part) 33 and the upper part 32 whenthe rolling unit 33 rolls. In addition, the adjunct part 101, which isintroduced in order to give reaction force and tactile feeling to afinger, bends to generate a restoration force and, by touching theundersurface of a finger, the feeling of contact. The lubricant coatingtechnique Teflon (registered trademark) can be applied to the surface ofthe rolling unit 33 in contact with the touch panel 2 to make it slidesmoothly on a touch panel surface.

Another prototype of the touch-panel-input-assisting-device of thisinvention is shown in FIG. 10. This touch-panel-input-assisting-deviceis made by integral molding as a single piece of a rubber or conductiverubber material. The part 33 of it operates as the rolling unit(contact-pattern-generation-means), the holding part (holding means) 39is where it is held by fingers, the adjunct part 101 gives tactilefeeling when it contacts a finger and is crushed, and the part 102 isthe projection to contact the a touch panel. The figures in (a), (b),and (c) show the touch-panel-input-assisting-device seen from differentviewpoints. The figures in (d), (e), and (f) show how the rolling unit33 rolls and the contact position moves starting from P1, moving to P2and P3 on the touch panel surface when the fingers apply a force (F)from the direction approximately perpendicular to the touch panelsurface and approach the touch panel. The rubber material constitutingthe device is deformed and gives a reaction force to a finger when ittries to restore its original shape due to its elasticity, and this canbe the reaction-force-generation-means. Moreover, the projection part102 would contact a touch panel and the contact pattern will changesuddenly if the finger holding the holding means 39 moves very close tothe touch panel and the rolling unit rolls on the touch panel as shownin (f). The program can detect this sudden change in the contact patternand judge that the finger moved to the specified position.

The flow chart of the tactile sense interlocking program is shown inFIG. 9. The contact-pattern-generation-means of thetouch-panel-input-assisting-device of this invention converts variousfinger movements into rolling movements of the rolling unit on the touchpanel. The pattern of movement or speed of the contact point between therolling unit and the touch panel must take a form that is characteristicto each pattern of finger movement. Meanwhile, tactile feedback (tactilefeeling) such as a resistance force, reaction force, vibration, or afeeling of unevenness is generated by thereaction-force-generation-means and given to a finger in synchronizingwith the specific feature of the contact pattern or the finger movement.With this tactile feedback synchronized with the contact pattern, anoperator can control his finger movement with high precision. On theother hand, the tactile sense interlocking program of this inventionanalyzes the movement pattern or speed pattern of the contact positionof a rolling unit and a touch panel to find the characteristic form,estimate the original finger movement, and it performs specificoperation in synchronizing with the finger movement or the tactilefeedback the finger perceives.

For example, when a rolling unit separates from a touch panel, thecontact position of a rolling unit and a panel moves for a specificdistance or arrives at a specified position, the movement or speedpattern of the contact position takes a specific pattern, the contactposition suddenly stops, or the contact position suddenly reverses itsmoving direction, the operation specific to each event is performed,synchronizing with the event. The flow chart of FIG. 9 shows an easy andconcrete example to explain the procedure for the synchronizationmentioned above. Each variable in the program is initialized first andthe procedure is started. Chattering process is performed to the signalabout the contact state of a touch panel during the procedures, andshort-time disconnection is ignored and dealt as a continuous contact.

After performing the chattering process, in Step 1, it is detectedwhether the rolling unit touches the touch panel. When the rolling unittouches the touch panel and it turns out that it is not in contact atthe last time by Step 4, since it means it contacts for the first time,it is judged as the contact to have started, the time of a timer isreset to 0, and the information on a contact position and time isrecorded by Step 5. And monitor of the contact state and coordinates ofthe contact position is continued. Moreover, if it is in contact at thelast time, it considers that the contact has been continuing and thespeed and average speed of the contact position is computed in Step 6.

And in Step 7, it is evaluated whether, the contact position of arolling unit and a touch panel arrived at the specific position, ormoved for a specific distance, or its speed exceeded a specificthreshold, or the contact position stopped after exceeding the speed ofa certain size, or its moving direction was reversed. And if either ofthese conditions or a combination of some of them is satisfied, in Step8, a predetermined operation is performed according to those conditions.The value of variable used in the operation is defined according to theaverage speed, the maximum speed, or the amount of movement of thecontact position. Various finger movements can be estimated from themovement pattern of the contact position. For example, if the contactposition stopped after moving with the speed exceeding a threshold andthen started to move to the opposite direction, it is estimated that thefinger approached the touch panel, contacted its surface, stood still onthe touch panel, and separated from it. Although various kinds of“predetermined or specific” value has been used in the aboveexplanation, such predetermined or specific value is a part of the datafor synchronization and should be determined depending on thecharacteristic of the touch-panel-input-assisting-device. These valuesshould be memorized in the memory of a computer or embedded in theprocedure of the program in advance, and used in the tactile senseinterlocking program in order to take the synchronization between thefinger movement or the tactile feeling the finger perceives and theprogram.

Thus, the movement of a finger or the reaction force given to a fingeris estimated from the moving pattern or the speed pattern of the contactposition, and a predetermined operation is performed in a predeterminedquantity according to it. If it is detected in Step 21 that the rollingunit touched the touch panel at the last time, but it is not in contactat present, then it is judged that the rolling unit separated from thepanel. Then, the position at the time of separation is recorded in Step22, and also the sound is stopped in Step 24. This is going to imitatethe real piano in which the sound continues while pushing a key, but thesound declines immediately if a finger is lifted from the key. Finallyin Step 25, after resetting all the variables, it returns to Step 1 andthe same procedure is repeated.

(Correspondence Between the Name of Each Means in the Claim, and theComponent of Each Figure of an Enforcement Form)

-   The rolling unit 31 of FIG. 2 is equivalent to the    contact-pattern-generation-means.-   The silicone rubber sheet 50 and the support component 53 of FIG. 2    are equivalent to the wearing means.-   The metal connector 45, the spring 44, the joint component 41, and    the silicone rubber sheet 50 of FIG. 2 are equivalent to a    reaction-force-generation-means.-   The rolling unit 31 of FIG. 3 and FIG. 4 is equivalent to a    contact-pattern-generation-means.-   The bands 42 a, 42 b, and 42 c to tie around the finger and the    buckle 43 of FIG. 3 and FIG. 4 are equivalent to the wearing means.-   The bands 42 a and 42 c to tie around a finger, the joint component    41, the spring 44, and the metal connector 45 of FIG. 3 and FIG. 4    are equivalent to the reaction-force-generation-means.-   The rolling unit 31 of FIG. 5 is equivalent to the    contact-pattern-generation-means.-   The silicone rubber sheet 50 of FIG. 5 is equivalent to the wearing    means.-   The silicone rubber cord 77 and the silicone rubber sheet 50 of FIG.    5 are equivalent to the reaction-force-generation-means.-   The rolling unit 33 of FIG. 6 is equivalent to the    contact-pattern-generation-means.-   The upper part 32 and the side part 34 of FIG. 6 are equivalent to    the wearing means.-   The axis of rotation 35, the upper part 32, and the adjunct part 101    of FIG. 6 are equivalent to the reaction-force-generation-means.-   The rolling unit 33 and the projection part 102 of FIG. 10 are    equivalent to the contact-pattern-generation-means.-   The reference numeral 39 of FIG. 10 is the holding means. The    adjunct part 101 of FIG. 10, the rolling unit 33, and the holding    means 34 of FIG. 6 are equivalent to the    reaction-force-generation-means.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of atouch-panel-input-assisting-device and a computer system using a tactilesense interlocking program by the form of implementation of thisinvention.

FIG. 2 is a side sectional view showing atouch-panel-input-assisting-device. The spherical rolling unit can berolled symmetrically by the same strength of the force to any directionof front and rear, right and left. The reaction force is also generatedto be isotropic. This isotropic nature is realized by using only onespring which draws the spherical rolling unit to the elastic sheetstretched over the touch panel. (a) shows the neutral state, (b) showsthe scene of the rolling unit pushed downward with the finger, and (c)shows the scene of the rolling unit rolled.

FIG. 3 is a figure showing the type worn by a finger among thetouch-panel-input-assisting-device by the form of implementation of thisinvention. (a) is the side view and (b) is the view looked at from thefront of a finger. The touch-panel-input-assisting-device is equippedwith the band.

FIG. 4 is a figure showing the scenes that the rolling unit of thefinger wearing type shown in FIG. 3 is made to roll on the touch panel.(a) shows the scene of the rolling unit contacting the touch panelbefore rolling, (b) shows the side view corresponding to (a), (c) is theview from the front of a finger when the rolling unit rolls on the touchpane. (d) shows the side view of the rolling unit when it is made toroll in another direction.

FIG. 5 is a figure showing how to produce thetouch-panel-input-assisting-device shown in FIG. 2 at a low cost usingthe silicone rubber cord unified with the sheet instead of the spring.

FIG. 6 is a figure showing that the finger movement approaching a touchpanel is converted into the rolling movement of a rolling unit. Thetouch-panel-input-assisting-device of the finger equipped (holding) typeis shown. The contact position moves as the rolling unit rolls. Thescene of the rolling unit contacting the panel, the scene of the fingerapproaching the panel moderately, and the scene of the fingerapproaching the panel to the limit and obtained a feeling of acollision, are shown in (a), (b) and (c) respectively. The single pieceversion of the touch-panel-input-assisting-device is shown in (d), (e)and (f) which operates by the same principle as (a), (b) and (c).

FIG. 7 is a figure showing a means to detect the amount of deformationwhen pressing and deforming a rolling unit with a finger. The degree ofdeformation is detected by measuring the size of contact area. (a) showsthe original shape and (b) shows the deformed shape.

FIG. 8 is a figure showing a scene displayed on the screen in a game inwhich a bat hits a ball.

FIG. 9 is a flow chart showing an example of the tactile senseinterlocking program.

FIG. 10 is a figure showing another example (FIG. 10( a)-(c)) of thetouch-panel-input-assisting-device and the diagrams (FIG. 10( d)-(f)) toexplain the operation of this invention.

FIG. 11 is a block diagram explaining the feature of the computeroperation method or the tactile sense interlocking program of thisinvention.

FIG. 12 is a figure explaining the reason for defining the data forsynchronization depending on the characteristic of thetouch-panel-input-assisting-device.

1. A touch panel input assisting device comprising: acontact-pattern-generation-means interposed between a touch panelequipped with a contact position detection means and a finger operatingthe touch panel, the contact-pattern-generation-means changing a contactpattern with the touch panel so that the contact pattern is interlockedwith a finger movement; a wearing-means for detachably equipping withthe touch panel or the finger, or a holding-means for holding with thefinger; and a reaction-force-generation-means for applying to the fingera reaction force which changes synchronizing with a change of thecontact pattern.
 2. The touch panel input assisting device according toclaim 1, wherein the contact-pattern-generation-means generates thecontact pattern through a rolling unit which rolls on the touch panelaccording to the finger movement and changes its contact position on thetouch panel, and the reaction-force-generation-means generates thereaction force by using a restoration force of an elastic componentwhich is interposed between the rolling unit and the touch panel or thefinger.
 3. The touch panel input assisting device according to claim 1,wherein the contact-pattern-generation-means changes the contact patternwith the touch panel so that the contact pattern is interlocked with thefinger movement approximately perpendicular to a surface of the touchpanel.
 4. The touch panel input assisting device according to claim 1,wherein the contact-pattern-generation-means and the wearing means aremade from a conductive and elastic material.
 5. The touch panel inputassisting device according to any one of claims 1 to 4, wherein thecontact-pattern-generation-means is constituted with a component whichslips smoothly on a surface of the touch panel.
 6. A computer operatingmethod for synchronizing an operation of a computer with a reactionforce which an touch panel input assisting device gives to a finger,wherein the device comprises a contact-pattern-generation-meansinterposed between a touch panel equipped with a contact positiondetection means and a finger operating the touch panel, thecontact-pattern-generation-means changing a contact pattern with thetouch panel so that the contact pattern is interlocked with a fingermovement; a wearing-means for detachably equipping with the touch panelor the finger, or a holding-means for holding with the finger; and areaction-force-generation-means for applying to the finger the reactionforce which changes synchronizing with a change of the contact pattern,wherein the method synchronizes the operation of the computer with thereaction force which the device gives to the finger by usingdata-for-synchronization which are defined in advance depending on thecharacteristics of the device and memorized in a storage medium insidethe computer and by using a mapping procedure which maps the contactpattern inputted through the touch panel connected to the computer tothe operation of the computer.
 7. A computer-executable, tactile senseinterlocking program for synchronizing an operation of a computer with areaction force which an touch panel input assisting device gives to afinger, wherein the device comprises a contact-pattern-generation-meansinterposed between a touch panel equipped with a contact positiondetection means and a finger operating the touch panel, thecontact-pattern-generation-means changing a contact pattern with thetouch panel so that the contact pattern is interlocked with a fingermovement; a wearing-means for detachably equipping with the touch panelor the finger, or a holding-means for holding with the finger; and areaction-force-generation-means for applying to the finger the reactionforce which changes synchronizing with a change of the contact pattern,wherein the program synchronizes the operation of the computer with thereaction force which the device gives to the finger by usingdata-for-synchronization which are defined in advance depending on thecharacteristics of the device and memorized in a storage medium insidethe computer and by using a mapping procedure which maps the contactpattern inputted through the touch panel connected to the computer tothe operation of the computer.